Method and apparatus for last coefficient indexing for high efficiency video coding

ABSTRACT

A method, apparatus, article of manufacture, and a memory structure for encoding and decoding transform unit coefficients is disclosed. In one embodiment, the encoding of the transform unit coefficients is performed by determining if all of the coefficients of a portion of the transform unit disposed on a portion of a scan pattern are zero, setting a flag associated with the portion of the transform unit to a first value if at least one of the coefficients of the portion of the transform unit coefficients disposed on the portion of the scan pattern are non-zero, and setting the flag associated with the portion of the transform unit coefficients to a second value if all of the coefficients of the portion of the transform unit coefficients disposed in the portion of the scan pattern are zero.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims benefit of U.S. Provisional Patent ApplicationNo. 61/577,898, entitled “LAST COEFFICIENT GROUP POSITION CODING INHEVC,” by Xue Fang, Krit Panusopone, and Limin Wang, filed Dec. 20,2011, which application is also hereby incorporated by reference herein.

This application is related to U.S. Utility patent application Ser. No.13/722,702, entitled “METHOD AND APPARATUS FOR LAST COEFFICIENT GROUPPOSITION CODING,” by Xue Fang, Krit Panusopone, and Limin Wang, filed onDec. 20, 2012 which is hereby incorporated by reference herein.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to systems and methods for encoding data,and in particular to a system and method for storing motion andretrieving data associated with high efficiency video coded data.

2. Description of the Related Art

There is rapid growth in the technologies associated with thegeneration, transmission, and reproduction of media programs. Thesetechnologies include coding schemes that permit digital versions of themedia programs to be encoded to compress them to much smaller size andfacilitate their transmission, storage, reception and playback. Thesetechnologies have application in personal video recorders (PVRs), videoon demand (VOD), multiple channel media program offerings,interactivity, mobile telephony, and media program transmission.

Without compression, digital media programs are typically too large totransmit and/or store for a commercially acceptable cost. However,compression of such programs has made the transmission and storage ofsuch digital media programs not only commercially feasible, butcommonplace.

Initially, the transmission of media programs involved low to mediumresolution images transmitted over high bandwidth transmission mediasuch as cable television and satellite. However, such transmission hasevolved to include lower bandwidth transmission media such as Internettransmission to fixed and mobile devices via computer networks, WiFi,Mobile TV and third and fourth generation (3G and 4G) networks. Further,such transmissions have also evolved to include high definition mediaprograms such as high definition television (HDTV), which havesignificant transmission bandwidth and storage requirements.

The encoding and decoding of compressed video signals typically involvesthe storage and retrieval of large volumes of data by both the encoderof the signals and the decoder of the signals. As a consequence, suchencoders and decoders require increased storage and processing capacity.What is needed is a system and method that reduces the storage andprocessing required in the encoder and decoder. The present inventionsatisfies this need.

SUMMARY OF THE INVENTION

To address the requirements described above, this document discloses amethod, apparatus, article of manufacture, and a memory structure forprocessing a transform unit having an array of coefficients isdisclosed. One embodiment is evidenced by a method comprisingdetermining if all of the coefficients of a portion of the transformunit disposed on a portion of a scan pattern are zero, setting a flagassociated with the portion of the transform unit to a first value if atleast one of the coefficients of the portion of the transform unitcoefficients disposed on the portion of the scan pattern are non-zero,and setting the flag associated with the portion of the transform unitcoefficients to a second value if all of the coefficients of the portionof the transform unit coefficients disposed in the portion of the scanpattern are zero.

Another embodiment is manifested by an apparatus comprising a processorand a memory coupled to the processor, storing instructions forperforming steps comprising the operations listed above.

The features, functions, and advantages that have been discussed can beachieved independently in various embodiments of the present inventionor may be combined in yet other embodiments further details of which canbe seen with reference to the following description and drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

Referring now to the drawings in which like reference numbers representcorresponding parts throughout:

FIG. 1 is a diagram depicting an exemplary embodiment of a videocoding-decoding system that can be used for transmission and/or storageand retrieval of audio and/or video information;

FIG. 2A is a diagram of one embodiment of a codec system 200A in whichthe encoded AV information is transmitted to and received at anotherlocation;

FIG. 2B is a diagram depicting an exemplary embodiment of codec systemin which the encoded information is stored and later retrieved forpresentation, hereinafter referred to as codec storage system;

FIG. 2C is a diagram depicting an exemplary content distribution systemcomprising an encoder and a decoder that can be used to transmit andreceive HEVC data;

FIG. 3 is a block diagram illustrating one embodiment of the sourceencoder;

FIG. 4 is a diagram depicting a picture of AV information, such as oneof the pictures in the picture sequence;

FIG. 5 is a diagram showing an exemplary partition of a coding treeblock into coding units;

FIG. 6 is a diagram illustrating a representation of a representativequadtree and data parameters for the code tree block partitioning shownin FIG. 5;

FIG. 7 is a diagram illustrating the partition of a coding unit into oneor more prediction units;

FIG. 8 is a diagram showing a coding unit partitioned into fourprediction units and an associated set of transform units;

FIG. 9 is a diagram showing RQT codetree for the transform unitsassociated with the coding unit in the example of FIG. 8;

FIG. 10 is a diagram illustrating spatial prediction of predictionunits;

FIG. 11 is a diagram illustrating temporal prediction;

FIG. 12 is a diagram illustrating the use of motion vector predictors(MVPs);

FIG. 13 is a diagram illustrating exemplary scan patterns used forcoding of the quantized coefficients of a non-square transform unit;

FIG. 14A is an diagram of an exemplary transform unit coefficient map;

FIG. 14B is a diagram of an exemplary significance map of the transformunit of FIG. 14A;

FIG. 15A presents a table for determining the initialization variablesin an entropy encoder;

FIG. 15B presents a table showing the an offset for different codingunit type;

FIG. 16A is diagram presenting an exemplary technique for performing alast coefficient coding technique that can be used in an HEVCencoding/decoding system;

FIG. 16B is a diagram illustrating the decoding of the encodedbitstream, using the last coefficient coding described in FIG. 16A;

FIG. 17 is a diagram illustrating the operations of FIGS. 16A and 16B;

FIG. 18A is a diagram of 16×16 transform unit;

FIG. 18B is a significance map generated from the transform unitillustrated in FIG. 18A;

FIG. 18C is a diagram illustrating an exemplary array scan pattern and alast significant coefficient of a transform unit;

FIG. 18D is a diagram illustrating a transform unit that has beenpartitioned into sixteen non-overlapping subblocks;

FIG. 18E is a diagram of an L1 significance map corresponding to thepartitioned transform unit of FIG. 18D;

FIG. 18F is a diagram of an alternative L1 significance map showingencoded subblock values;

FIG. 18G is a diagram illustrating the coding of the transformcoefficients within an exemplary subblock;

FIG. 19 is diagram presenting an exemplary technique for performing alast subblock coding technique that can be used in an HEVCencoding/decoding system;

FIG. 20 presents a diagram illustrating the coefficients of a candidatesubblock having the first encoded value indicating that it includesnon-zero coefficients;

FIG. 21 is a diagram of the last significant subblock of the transformunit;

FIG. 22 is a diagram illustrating an exemplary technique for decodingtransform unit coefficients that were encoded using the foregoing lastcoefficient group position technique; and

FIG. 23 illustrates an exemplary processing system that can be used toimplement the embodiments of the invention.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

In the following description, reference is made to the accompanyingdrawings which form a part hereof, and which is shown, by way ofillustration, several embodiments of the present invention. It isunderstood that other embodiments may be utilized and structural changesmay be made without departing from the scope of the present invention.

Audio-Visual Information Transception and Storage

FIG. 1 is a diagram depicting an exemplary embodiment of a videocoding-decoding (codec) system 100 that can be used for transmissionand/or storage and retrieval of audio and/or video information. Thecodec system 100 comprises an encoding system 104, which acceptsaudio-visual (AV) information 102 and processes the AV information 102to generate encoded (compressed) AV information 106, and a decodingsystem 112, which processes the encoded AV information 106 to producerecovered AV information 114. Since the encoding and decoding processesare not lossless, the recovered AV information 114 is not identical tothe initial AV information 102, but with judicious selection of theencoding processes and parameters, the differences between the recoveredAV information 114 and the unprocessed AV information 102 are acceptableto human perception.

The encoded AV information 106 is typically transmitted or stored andretrieved before decoding and presentation, as performed by transception(transmission and reception) or storage/retrieval system 108.Transception losses may be significant, but storage/retrieval losses aretypically minimal or non-existent, hence, the transcepted AV information110 provided to the decoding system 112 is typically the same as orsubstantially the same as the encoded AV information 106.

FIG. 2A is a diagram of one embodiment of a codec system 200A in whichthe encoded AV information 106 is transmitted to and received at anotherlocation. A transmission segment 230 converts an input AV information102 into a signal appropriate for transmission and transmits theconverted signal over the transmission channel 212 to the receptionsegment 232. The reception segment 232 receives the transmitted signal,and converts the received signal into the recovered AV information 114for presentation. As described above, due to coding and transmissionlosses and errors, the recovered AV information 114 may be of lowerquality than the AV information 102 that was provided to thetransmission segment 230. However, error correcting systems may beincluded to reduce or eliminate such errors. For example, the encoded AVinformation 106 may be forward error correction (FEC) encoded by addingredundant information, and such redundant information can be used toidentify and eliminate errors in the reception segment 230.

The transmission segment 102 comprises one or more source encoders 202to encode multiple sources of AV information 102. The source encoder 202encodes the AV information 102 primarily for purposes of compression toproduce the encoded AV information 106, and may include, for example aprocessor and related memory storing instructions implementing a codecsuch as MPEG-1, MPEG-2, MPEG-4 AVC/H.264, HEVC or similar codec, asdescribed further below.

The codec system 200A may also include optional elements indicated bythe dashed lines in FIG. 2A. These optional elements include a videomultiplex encoder 204, an encoding controller 208, and a videodemultiplexing decoder 218. The optional video multiplex encoder 204multiplexes encoded AV information 106 from an associated plurality ofsource encoder(s) 202 according to one or more parameters supplied bythe optional encoding controller 208. Such multiplexing is typicallyaccomplished in the time domain and is data packet based.

In one embodiment, the video multiplex encoder 204 comprises astatistical multiplexer, which combines the encoded AV information 106from a plurality of source encoders 202 so as to minimize the bandwidthrequired for transmission. This is possible, since the instantaneous bitrate of the coded AV information 106 from each source encoder 202 canvary greatly with time according to the content of the AV information102. For example, scenes having a great deal of detail and motion (e.g.sporting events) are typically encoded at higher bitrates than sceneswith little motion or detail (e.g. portrait dialog). Since each sourceencoder 202 may produce information with a high instantaneous bit ratewhile another source encoder 202 produces information with a lowinstantaneous bit rate, and since the encoding controller 208 cancommand the source encoders 202 to encode the AV information 106according to certain performance parameters that affect theinstantaneous bit rate, the signals from each of the source encoders 106(each having a temporally varying instantaneous bit rate) can becombined together in an optimal way to minimize the instantaneous bitrate of the multiplexed stream 205.

As described above, the source encoder 202 and the video multiplex coder204 may optionally be controlled by a coding controller 208 to minimizethe instantaneous bit rate of the combined video signal. In oneembodiment, this is accomplished using information from a transmissionbuffer 206 which temporarily stores the coded video signal and canindicate the fullness of the buffer 206. This allows the codingperformed at the source encoder 202 or video multiplex coder 204 to be afunction of the storage remaining in the transmission buffer 206.

The transmission segment 230 also may comprise a transmission encoder210, which further encodes the video signal for transmission to thereception segment 232. Transmission encoding may include for example,the aforementioned FEC coding and/or coding into a multiplexing schemefor the transmission medium of choice. For example, if the transmissionis by satellite or terrestrial transmitters, the transmission encoder114 may encode the signal into a signal constellation beforetransmission via quadrature amplitude modulation (QAM) or similarmodulation technique. Also, if the encoded video signal is to bestreamed via an Internet protocol device and the Internet, thetransmission encodes the signal according to the appropriate protocol.Further, if the encoded signal is to be transmitted via mobiletelephony, the appropriate coding protocol is used, as further describedbelow.

The reception segment 232 comprises a transmission decoder 214 toreceive the signal that was coded by the transmission coder 210 using adecoding scheme complementary to the coding scheme used in thetransmission encoder 214. The decoded received signal may be temporarilystored by optional reception buffer 216, and if the received signalcomprises multiple video signals, the received signal is multiplexdecoded by video multiplex decoder 218 to extract the video signal ofinterest from the video signals multiplexed by the video multiplex coder204. Finally, the video signal of interest is decoded by source decoder220 using a decoding scheme or codec complementary to the codec used bythe source encoder 202 to encode the AV information 102.

In one embodiment, the transmitted data comprises a packetized videostream transmitted from a server (representing the transmitting segment230) to a client (representing the receiving segment 232). In this case,the transmission encoder 210 may packetize the data and embed networkabstract layer (NAL) units in network packets. NAL units define a datacontainer that has header and coded elements, and may correspond to avideo frame or other slice of video data.

The compressed data to be transmitted may packetized and transmitted viatransmission channel 212, which may include a Wide Area Network (WAN) ora Local Area Network (LAN). Such a network may comprise, for example, awireless network such as Wi-Fi, an Ethernet network, an Internet networkor a mixed network composed of several different networks. Suchcommunication may be affected via a communication protocol, for exampleReal-time Transport Protocol (RTP), User Datagram Protocol (UDP) or anyother type of communication protocol. Different packetization methodsmay be used for each network abstract layer (NAL) unit of the bitstream.In one case, one NAL unit size is smaller than the maximum transportunit (MTU) size corresponding to the largest packet size that can betransmitted over the network without being fragmented. In this case, theNAL unit is embedded into a single network packet. In another case,multiple entire NAL units are included in a single network packet. In athird case, one NAL unit may be too large to be transmitted in a singlenetwork packet and is thus split into several fragmented NAL units witheach fragmented NAL unit being transmitted in an individual networkpacket. Fragmented NAL unit are typically sent consecutively fordecoding purposes.

The reception segment 232 receives the packetized data and reconstitutesthe NAL units from the network packet. For fragmented NAL units, theclient concatenates the data from the fragmented NAL units in order toreconstruct the original NAL unit. The client 232 decodes the receivedand reconstructed data stream and reproduces the video images on adisplay device and the audio data by a loud speaker.

FIG. 2B is a diagram depicting an exemplary embodiment of codec systemin which the encoded information is stored and later retrieved forpresentation, hereinafter referred to as codec storage system 200B. Thisembodiment may be used, for example, to locally store information in adigital video recorder (DVR), a flash drive, hard drive, or similardevice. In this embodiment, the AV information 102 is source encoded bysource encoder 202, optionally buffered by storage buffer 234 beforestorage in a storage device 236. The storage device 236 may store thevideo signal temporarily or for an extended period of time, and maycomprise a hard drive, flash drive, RAM or ROM. The stored AVinformation is then retrieved, optionally buffered by retrieve buffer238 and decoded by the source decoder 220.

FIG. 2C is another diagram depicting an exemplary content distributionsystem 200C comprising a coding system or encoder 202 and a decodingsystem or decoder 220 that can be used to transmit and receive HEVCdata. In some embodiments, the coding system 202 can comprise an inputinterface 256, a controller 241 a counter 242 a frame memory 243, anencoding unit 244, a transmitter buffer 267 and an output interface 257.The decoding system 220 can comprise a receiver buffer 259, a decodingunit 260, a frame memory 261 and a controller 267. The coding system 202and the decoding system 220 can be coupled with each other via atransmission path which can carry a compressed bit stream. Thecontroller 241 of the coding system 202 can control the amount of datato be transmitted on the basis of the capacity of the transmitter buffer267 or receiver buffer 259 and can include other parameters such as theamount of data per a unit of time. The controller 241 can control theencoding unit 244 to prevent the occurrence of a failure of a receivedsignal decoding operation of the decoding system 220. The controller 241can be a processor or include, by way of a non-limiting example, amicrocomputer having a processor, a random access memory and a read onlymemory.

Source pictures 246 supplied from, by way of a non-limiting example, acontent provider can include a video sequence of frames including sourcepictures in a video sequence. The source pictures 246 can beuncompressed or compressed. If the source pictures 246 are uncompressed,the coding system 202 can have an encoding function. If the sourcepictures 246 are compressed, the coding system 202 can have atranscoding function. Coding units can be derived from the sourcepictures utilizing the controller 241. The frame memory 243 can have afirst area that can be used for storing the incoming frames from thesource pictures 246 and a second area that can be used for reading outthe frames and outputting them to the encoding unit 244. The controller241 can output an area switching control signal 249 to the frame memory243. The area switching control signal 249 can indicate whether thefirst area or the second area is to be utilized.

The controller 241 can output an encoding control signal 250 to theencoding unit 244. The encoding control signal 250 can cause theencoding unit 202 to start an encoding operation, such as preparing theCoding Units based on a source picture. In response to the encodingcontrol signal 250 from the controller 241, the encoding unit 244 canbegin to read out the prepared Coding Units to a high-efficiencyencoding process, such as a prediction coding process or a transformcoding process which process the prepared Coding Units generating videocompression data based on the source pictures associated with the CodingUnits.

The encoding unit 244 can package the generated video compression datain a packetized elementary stream (PES) including video packets. Theencoding unit 244 can map the video packets into an encoded video signal248 using control information and a program time stamp (PTS) and theencoded video signal 248 can be transmitted to the transmitter buffer267.

The encoded video signal 248, including the generated video compressiondata, can be stored in the transmitter buffer 267. The informationamount counter 242 can be incremented to indicate the total amount ofdata in the transmitter buffer 267. As data is retrieved and removedfrom the buffer, the counter 242 can be decremented to reflect theamount of data in the transmitter buffer 267. The occupied areainformation signal 253 can be transmitted to the counter 242 to indicatewhether data from the encoding unit 244 has been added or removed fromthe transmitter buffer 267 so the counter 242 can be incremented ordecremented. The controller 241 can control the production of videopackets produced by the encoding unit 244 on the basis of the occupiedarea information 253 which can be communicated in order to anticipate,avoid, prevent, and/or detect an overflow or underflow from taking placein the transmitter buffer 267.

The information amount counter 242 can be reset in response to a presetsignal 254 generated and output by the controller 241. After theinformation amount counter 242 is reset, it can count data output by theencoding unit 244 and obtain the amount of video compression data and/orvideo packets which have been generated. The information amount counter242 can supply the controller 241 with an information amount signal 255representative of the obtained amount of information. The controller 241can control the encoding unit 244 so that there is no overflow at thetransmitter buffer 267.

In some embodiments, the decoding system 220 can comprise an inputinterface 266, a receiver buffer 259, a controller 267, a frame memory261, a decoding unit 260 and an output interface 267. The receiverbuffer 259 of the decoding system 220 can temporarily store thecompressed bit stream, including the received video compression data andvideo packets based on the source pictures from the source pictures 246.The decoding system 220 can read the control information andpresentation time stamp information associated with video packets in thereceived data and output a frame number signal 263 which can be appliedto the controller 220. The controller 267 can supervise the countednumber of frames at a predetermined interval. By way of a non-limitingexample, the controller 267 can supervise the counted number of frameseach time the decoding unit 260 completes a decoding operation.

In some embodiments, when the frame number signal 263 indicates thereceiver buffer 259 is at a predetermined capacity, the controller 267can output a decoding start signal 264 to the decoding unit 260. Whenthe frame number signal 263 indicates the receiver buffer 259 is at lessthan a predetermined capacity, the controller 267 can wait for theoccurrence of a situation in which the counted number of frames becomesequal to the predetermined amount. The controller 267 can output thedecoding start signal 263 when the situation occurs. By way of anon-limiting example, the controller 267 can output the decoding startsignal 264 when the frame number signal 263 indicates the receiverbuffer 259 is at the predetermined capacity. The encoded video packetsand video compression data can be decoded in a monotonic order (i.e.,increasing or decreasing) based on presentation time stamps associatedwith the encoded video packets.

In response to the decoding start signal 264, the decoding unit 260 candecode data amounting to one picture associated with a frame andcompressed video data associated with the picture associated with videopackets from the receiver buffer 259. The decoding unit 260 can write adecoded video signal 269 into the frame memory 261. The frame memory 261can have a first area into which the decoded video signal is written,and a second area used for reading out decoded pictures 262 to theoutput interface 267.

In various embodiments, the coding system 202 can be incorporated orotherwise associated with a transcoder or an encoding apparatus at aheadend and the decoding system 220 can be incorporated or otherwiseassociated with a downstream device, such as a mobile device, a set topbox or a transcoder.

Source Encoding/Decoding

As described above, the encoders 202 employ compression algorithms togenerate bit streams and/or files of smaller size than the originalvideo sequences in the AV information 102. Such compression is madepossible by reducing spatial and temporal redundancies in the originalsequences.

Prior art encoders 202 include those compliant with the videocompression standard H.264/MPEG-4 AVC (“Advanced Video Coding”)developed by between the “Video Coding Expert Group” (VCEG) of the ITUand the “Moving Picture Experts Group” (MPEG) of the ISO, in particularin the form of the publication “Advanced Video Coding for GenericAudiovisual Services” (March 2005), which is hereby incorporated byreference herein.

HEVC “High Efficiency Video Coding” (sometimes known as H.265) isexpected to replace the H.264/MPEG-4 AVC. HEVC introduces new codingtools and entities that are generalizations of the coding entitiesdefined in H.264/AVC, as further described below, and as described inU.S. Provisional Application Nos. 61/562,418, 61/619,916, and61/595,141.

FIG. 3 is a block diagram illustrating one embodiment of the sourceencoder 202. The source encoder 202 accepts AV information 102 and usessampler 302 sample the AV information 102 to produce a sequence 303 ofsuccessive of digital images or pictures, each having a plurality ofpixels. A picture can comprise a frame or a field, wherein a frame is acomplete image captured during a known time interval, and a field is theset of odd-numbered or even-numbered scanning lines composing a partialimage.

The sampler 302 produces an uncompressed picture sequence 303. Eachdigital picture can be represented by one or more matrices having aplurality of coefficients that represent information about the pixelsthat together comprise the picture. The value of a pixel can correspondto luminance or other information. In the case where several componentsare associated with each pixel (for example red-green-blue components orluminance-chrominance components), each of these components may beseparately processed.

Images can be segmented into “slices,” which may comprise a portion ofthe picture or may comprise the entire picture. In the H.264 standard,these slices are divided into coding entities called macroblocks(generally blocks of size 16 pixels×16 pixels) and each macroblock mayin turn be divided into different sizes of data blocks 102, for example4×4, 4×8, 8×4, 8×8, 8×16, 16×8. HEVC expands and generalizes the notionof the coding entity beyond that of the macroblock.

HEVC Coding Entities: CTU, CU, PU and TU

Like other video coding standards, HEVC is a block-based hybrid spatialand temporal predictive coding scheme. However, HEVC introduces newcoding entities that are not included with H.264/AVC standard. Thesecoding entities include (1) Coding tree block (CTUs), coding units(CUs), the predictive units (PUs) and transform units (TUs) and arefurther described below.

FIG. 4 is a diagram depicting a picture 400 of AV information 102, suchas one of the pictures in the picture sequence 303. The picture 400 isspatially divided into non-overlapping square blocks known as codingtree units(s), or CTUs 402. Unlike H.264 and previous video codingstandards where the basic coding unit is macroblock of 16×16 pixels, theCTU 402 is the basic coding unit of HEVC, and can be as large as 128×128pixels. As shown in FIG. 4, the CTUs 402 are typically referenced withinthe picture 400 in an order analogous to a progressive scan.

Each CTU 402 may in turn be iteratively divided into smaller variablesize coding units described by a “quadtree” decomposition furtherdescribed below. Coding units are regions formed in the image to whichsimilar encoding parameters are applied and transmitted in the bitstream314.

FIG. 5 is a diagram showing an exemplary partition of a CTU 402 intocoding units (CUs) such as coding unit 502A and 502B (hereinafteralternatively referred to as coding unit(s) 502). A single CTU 402 canbe divided into four CUs 502 such as CU 502A, each a quarter of the sizeof CTU 402. Each such divided CU 502A can be further divided into foursmaller CUs 502B of quarter size of initial CU 502A.

The division of CTUs 402 into CUs 502A and into smaller CUs 502B isdescribed by “quadtree” data parameters (e.g. flags or bits) that areencoded into the output bitstream 314 along with the encoded data asoverhead known as syntax.

FIG. 6 is a diagram illustrating a representation of a representativequadtree 600 and data parameters for the CTU 402 partitioning shown inFIG. 5. The quadtree 600 comprises a plurality of nodes including firstnode 602A at one hierarchical level and second node 602B at a lowerhierarchical level (hereinafter, quadtree nodes may be alternativelyreferred to as “nodes” 602). At each node 602 of a quadtree, a “splitflag” or bit “1” is assigned if the node 602 is further split intosub-nodes, otherwise a bit “0” is assigned.

For example, the CTU 402 partition illustrated in FIG. 5 can berepresented by the quadtree 600 presented in FIG. 6, which includes asplit flag of “1” associated with node 602A at the top CU 502 level(indicating there are 4 additional nodes at a lower hierarchical level).The illustrated quadtree 600 also includes a split flag of “1”associated with node 602B at the mid CU 502 level to indicate that thisCU is also partitioned into four further CUs 502 at the next (bottom) CUlevel. The source encoder 202 may restrict the minimum and maximum CU502 sizes, thus changing the maximum possible depth of the CU 502splitting.

The encoder 202 generates encoded AV information 106 in the form of abitstream 314 that includes a first portion having encoded data for theCUs 502 and a second portion that includes overhead known as syntaxelements. The encoded data includes data corresponding to the encodedCUs 502 (i.e. the encoded residuals together with their associatedmotion vectors, predictors, or related residuals as described furtherbelow). The second portion includes syntax elements that may representencoding parameters which do not directly correspond to the encoded dataof the blocks. For example, the syntax elements may comprise an addressand identification of the CU 502 in the image, a quantization parameter,an indication of the elected Inter/Intra coding mode, the quadtree 600or other information.

CUs 502 correspond to elementary coding elements and include two relatedsub-units: prediction units (PUs) and transform units (TUs), both ofwhich have a maximum size equal to the size of the corresponding CU 502.

FIG. 7 is a diagram illustrating the partition of a CU 502 into one ormore PUs 702. A PU 702 corresponds to a partitioned CU 502 and is usedto predict pixels values for intra-picture or inter-picture types. PUs702 are an extension of the partitioning of H.264/AVC for motionestimation, and are defined for each CU 502 that is not furthersubdivided into other CUs (“split flag”=0). At each leaf 604 of thequadtree 600, a final (bottom level) CU 502 of 2N×2N can possess one offour possible patterns of PUs: 2N×2N (702A), 2N×N (702B), N×2N (702C)and N×N (702D)), as shown in FIG. 7.

A CU 502 can be either spatially or temporally predictive coded. If a CU502 is coded in “intra” mode, each PU 702 of the CU 502 can have its ownspatial prediction direction and image information as further describedbelow. Also, in the “intra” mode, the PU 702 of the CU 502 may depend onanother CU 502 because it may use a spatial neighbor, which is inanother CU. If a CU 502 is coded in “inter” mode, each PU 702 of the CU502 can have its own motion vector(s) and associated referencepicture(s) as further described below.

FIG. 8 is a diagram showing a CU 502 partitioned into four PUs 702 andan associated set of transform units (TUs) 802. TUs 802 are used torepresent the elementary units that are spatially transformed by a DCT(Discrete Cosine Transform). The size and location of each blocktransform TU 802 within a CU 502 is described by a “residual” quadtree(RQT) further illustrated below.

FIG. 9 is a diagram showing RQT 900 for TUs 802 for the CU 502 in theexample of FIG. 8. Note that the “1” at the first node 902A of the RQT900 indicates that there are four branches and that the “1” at thesecond node 902B at the adjacent lower hierarchical level indicates thatthe indicated node further has four branches. The data describing theRQT 900 is also coded and transmitted as an overhead in the bitstream314.

The coding parameters of a video sequence may be stored in dedicated NALunits called parameter sets. Two types of parameter sets NAL units maybe employed. The first parameter set type is known as a SequenceParameter Set (SPS), and comprises a NAL unit that includes parametersthat are unchanged during the entire video sequence. Typically, an SPShandles the coding profile, the size of the video frames and otherparameters. The second type of parameter set is known as a PictureParameter Set (PPS), and codes different values that may change from oneimage to another.

Spatial and Temporal Prediction

One of the techniques used to compress a bitstream 314 is to forego thestorage of pixel values themselves and instead, predict the pixel valuesusing a process that can be repeated at the decoder 220 and store ortransmit the difference between the predicted pixel values and theactual pixel values (known as the residual). So long as the decoder 220can compute the same predicted pixel values from the informationprovided, the actual picture values can be recovered by adding theresiduals to the predicted values. The same technique can be used tocompress other data as well.

Referring back to FIG. 3, each PU 702 of the CU 502 being processed isprovided to a predictor module 307. The predictor module 307 predictsthe values of the PUs 702 based on information in nearby PUs 702 in thesame frame (intra-frame prediction, which is performed by the spatialpredictor 324) and information of PUs 702 in temporally proximate frames(inter-frame prediction, which is performed by the temporal predictor330). Temporal prediction, however, may not always be based on acollocated PU, since collocated PUs are defined to be located at areference/non-reference frame having the same x and y coordinates as thecurrent PU 702. These techniques take advantage of spatial and temporaldependencies between PUs 702.

Encoded units can therefore be categorized to include two types: (1)non-temporally predicted units and (2) temporally predicted units.Non-temporally predicted units are predicted using the current frame,including adjacent or nearby PUs 702 within the frame (e.g. intra-frameprediction), and are generated by the spatial predictor 324. Temporallypredicted units are predicted from one temporal picture (e.g. P-frames)or predicted from at least two reference pictures temporally aheadand/or behind (i.e. B-frames).

Spatial Prediction

FIG. 10 is a diagram illustrating spatial prediction of PUs 702. Apicture i may comprise a PU 702 and spatially proximate other PUs 1-4,including nearby PU 702N. The spatial predictor 324 predicts the currentblock (e.g. block C of FIG. 10) by means of an “intra-frame” predictionwhich uses PUs 702 of already-encoded other blocks of pixels of thecurrent image.

The spatial predictor 324 locates a nearby PU (e.g. PU 1, 2, 3 or 4 ofFIG. 10) that is appropriate for spatial coding and determines anangular prediction direction to that nearby PU. In HEVC, 35 directionscan be considered, so each PU may have one of 35 directions associatedwith it, including horizontal, vertical, 45 degree diagonal, 135 degreediagonal, DC etc. The spatial prediction direction of the PU isindicated in the syntax.

Referring back to the spatial predictor 324 of FIG. 3, this locatednearby PU is used to compute a residual PU 704 (e) as the differencebetween the pixels of the nearby PU 702N and the current PU 702, usingelement 305. The result is an intra-predicted PU element 1006 thatcomprises a prediction direction 1002 and the intra-predicted residualPU 1004. The prediction direction 1002 may be coded by inferring thedirection from spatially proximate PUs, and the spatial dependencies ofthe picture, enabling the coding rate of the intra prediction directionmode to be reduced.

Temporal Prediction

FIG. 11 is a diagram illustrating temporal prediction. Temporalprediction considers information from temporally neighboring pictures orframes, such as the previous picture, picture i−1.

Generally, temporal prediction includes single-prediction (P-type),which predicts the PU 702 by referring to one reference area from onlyone reference picture, and multiple prediction (B-type), which predictsthe PU by referring to two reference areas from one or two referencepictures. Reference images are images in the video sequence that havealready been coded and then reconstructed (by decoding).

The temporal predictor 330 identifies, in one or several of thesereference areas (one for P-type or several for B-type), areas of pixelsin a temporally nearby frame so that they can be used as predictors ofthis current PU 702. In the case where several areas predictors are used(B-type), they may be merged to generate one single prediction. Thereference area 1102 is identified in the reference frame by a motionvector (MV) 1104 that is defines the displacement between the current PU702 in current frame (picture i) and the reference area 1102 (refIdx) inthe reference frame (picture i−1). A PU in a B-picture may have up totwo MVs. Both MV and refIdx information are included in the syntax ofthe HEVC bitstream.

Referring again to FIG. 3, a difference between the pixel values betweenof the reference area 1102 and the current PU 702 may be computed byelement 305 as selected by switch 306. This difference is referred to asthe residual of the inter-predicted PU 1106. At the end of the temporalor inter-frame prediction process, the current PU 1006 is composed ofone motion vector MV 1104 and a residual 1106.

However, as described above, one technique for compressing data is togenerate predicted values for the data using means repeatable by thedecoder 220, computing the difference between the predicted and actualvalues of the data (the residual) and transmitting the residual fordecoding. So long as the decoder 220 can reproduce the predicted values,the residual values can be used to determine the actual values.

This technique can be applied to the MVs 1104 used in temporalprediction by generating a prediction of the MV 1104, computing adifference between the actual MV 1104 and the predicted MV 1104 (aresidual) and transmitting the MV residual in the bitstream 314. So longas the decoder 220 can reproduce the predicted MV 1104, the actual MV1104 can be computed from the residual. HEVC computes a predicted MV foreach PU 702 using the spatial correlation of movement between nearby PUs702.

FIG. 12 is a diagram illustrating the use of motion vector predictors(MVPs) in HEVC. Motion vector predictors V1, V2 and V3 are taken fromthe MVs 1104 of a plurality of blocks 1, 2, and 3 situated nearby oradjacent the block to encode (C). As these vectors refer to motionvectors of spatially neighboring blocks within the same temporal frameand can be used to predict the motion vector of the block to encode,these vectors are known as spatial motion predictors.

FIG. 12 also illustrates temporal motion vector predictor VT which isthe motion vector of the co-located block C′ in a previously decodedpicture (in decoding order) of the sequence (e.g. block of picture i−1located at the same spatial position as the block being coded (block Cof image i).

The components of the spatial motion vector predictors V1, V2 and V3 andthe temporal motion vector predictor VT can be used to generate a medianmotion vector predictor VM. In HEVC, the three spatial motion vectorpredictors may be taken as shown in FIG. 12, that is, from the blocksituated to the left of the block to encode (V1), the block situatedabove (V3) and from one of the blocks situated at the respective cornersof the block to encode (V2), according to a predetermined rule ofavailability. This MV predictor selection technique is known as AdvancedMotion Vector Prediction (AMVP).

A plurality of (typically five) MV predictor (MVP) candidates havingspatial predictors (e.g. V1, V2 and V3) and temporal predictor(s) VT istherefore obtained. In order to reduce the overhead of signaling themotion vector predictor in the bitstream, the set of motion vectorpredictors may reduced by eliminating data for duplicated motion vectors(for example, MVs which have the same value as other MVs may beeliminated from the candidates).

The encoder 202 may select a “best” motion vector predictor from amongthe candidates, and compute a motion vector predictor residual as adifference between the selected motion vector predictor and the actualmotion vector, and transmit the motion vector predictor residual in thebitstream 314. To perform this operation, the actual motion vector mustbe stored for later use by the decoder 220 (although it is nottransmitted in the bit stream 314. Signaling bits or flags are includedin the bitstream 314 to specify which MV residual was computed from thenormalized motion vector predictor, and are later used by the decoder torecover the motion vector. These bits or flags are further describedbelow.

Referring back to FIG. 3, the intra-predicted residuals 1004 and theinter-predicted residuals 1106 obtained from the spatial (intra) ortemporal (inter) prediction process are then transformed by transformmodule 308 into the transform units (TUs) 802 described above. A TU 802can be further split into smaller TUs using the RQT decompositiondescribed above with respect to FIG. 9. In HEVC, generally 2 or 3 levelsof decompositions are used and authorized transform sizes are from32×32, 16×16, 8×8 and 4×4. As described above, the transform is derivedaccording to a discrete cosine transform (DCT) or discrete sinetransform (DST).

The residual transformed coefficients are then quantized by quantizer310. Quantization plays a very important role in data compression. InHEVC, quantization converts the high precision transform coefficientsinto a finite number of possible values. Although the quantizationpermits a great deal of compression, quantization is a lossy operation,and the loss by quantization cannot be recovered.

The coefficients of the quantized transformed residual are then coded bymeans of an entropy coder 312 and then inserted into the compressed bitstream 310 as a part of the useful data coding the images of the AVinformation. Coding syntax elements may also be coded using spatialdependencies between syntax elements to increase the coding efficiency.HEVC offers context-adaptive binary arithmetic coding (CABAC). Otherforms or entropy or arithmetic coding may also be used.

In order to calculate the predictors used above, the encoder 202 decodesalready encoded PUs 702 using “decoding” loop 315, which includeselements 316, 318, 320, 322, 328. This decoding loop 315 reconstructsthe PUs and images from the quantized transformed residuals.

The quantized transform residual coefficients E are provided todequantizer 316, which applies the inverse operation to that ofquantizer 310 to produce dequantized transform coefficients of theresidual PU (E′) 708. The dequantized data 708 is then provided toinverse transformer 318 which applies the inverse of the transformapplied by the transform module 308 to generate reconstructed residualcoefficients of the PU (e′) 710.

The reconstructed coefficients of the residual PU 710 are then added tothe corresponding coefficients of the corresponding predicted PU (x′)702′ selected from the intra-predicted PU 1004 and the inter-predictedPU 1106 by selector 306. For example, if the reconstructed residualcomes from the “intra” coding process of the spatial predictor 324, the“intra” predictor (x′) is added to this residual in order to recover areconstructed PU (x″) 712 corresponding to the original PU 702 modifiedby the losses resulting from a transformation, for example in this casethe quantization operations. If the residual 710 comes from an “inter”coding process of the temporal predictor 330, the areas pointed to bythe current motion vectors (these areas belong to the reference imagesstored in reference buffer 328 referred by the current image indices)are merged then added to this decoded residual. In this way the originalPU 702 is modified by the losses resulting from the quantizationoperations.

To the extent that the encoder 202 uses motion vector predictiontechniques analogous to the image prediction techniques described above,the motion vector may be stored using motion vector buffer 329 for usein temporally subsequent frames. As further described below, a flag maybe set and transferred in the syntax to indicate that the motion vectorfor the currently decoded frame should be used for at least thesubsequently coded frame instead of replacing the contents of the MVbuffer 329 with the MV for the current frame.

A loop filter 322 is applied to the reconstructed signal (x″) 712 inorder to reduce the effects created by heavy quantization of theresiduals obtained, and to improve the signal quality. The loop filter322 may comprise, for example, a deblocking filter for smoothing bordersbetween PUs to visually attenuate high frequencies created by the codingprocess and a linear filter that is applied after all of the PUs for animage have been decoded to minimize the sum of the square difference(SSD) with the original image. The linear filtering process is performedon a frame by frame basis and uses several pixels around the pixel to befiltered, and also uses spatial dependencies between pixels of theframe. The linear filter coefficients may be coded and transmitted inone header of the bitstream typically a picture or slice header.

The filtered images, also known as reconstructed images, are then storedas reference images from reference image buffer 328 in order to allowthe subsequent “Inter” predictions taking place during the compressionof the subsequent images of the current video sequence.

Entropy Coding

The coding performed by coder 312 is performed by coding eachcoefficient of the quantized transform coefficients of the transformunit (TU) 802 obtained from the transformer 308 and quantizer 310. Suchcoding is typically performed for each transform coefficient or group ofcoefficients using one of a plurality of available scan patterns.

FIG. 13 is a diagram illustrating exemplary scan patterns used forcoding of the quantized coefficients of a non-square (4×6) TU 802. Thesescan patterns include a horizontal scan pattern 1302A (scan angle θ=0degrees), zig-zag, diagonal patterns 1302B and 1302D (scan angle θ=45and 135 degrees), and vertical scan 1302C (scan angle=270 degrees).Wavefront scanning (in which the encoder 312 uses information from thepreceding row of TUs 802, enabling parallel processing that may allowfor better compression than simple tiling) may be performed in anydirection.

The illustrated scan patterns begin at the TU 802 coefficient at theupper right of each TU 802 and ends at the lower right coefficient ofthe TU 802. However, the scan pattern may begin at the lower rightcoefficient 24 and end at the upper left coefficient 1, or for thatmatter, may follow a pattern from the innermost coefficient positions tothe outermost coefficient positions, following a clockwise (e.g. 10, 11,14, 15, 19, 18, 17, 13, 9, 5, 6, 7, 8, 12, 16, 20, 24, 23, 22, 21, 1, 2,3, 4) or counter-clockwise pattern or a pattern that follows theoutermost coefficient positions inward. Further, the scan pattern neednot be continuous (e.g. 10, 15, 14, 11, etc.). As will be describedfurther below, once the last significant (non-zero) coefficient positionis known, the coefficients can be decoded in the opposite direction ofthe scan pattern back to the first coefficient in the scan pattern.

FIG. 14A is a diagram of an exemplary TU 802 coefficient map 1402. Notethat many of the coefficients are zero coefficients (e.g. their value iszero). For each TU 802 and at each scanned position, a bit may beassigned or set to indicate if the corresponding coefficient is zero ornot, resulting in a significance map 1452 that is associated with eachTU 802.

FIG. 14B is a diagram showing an exemplary significance map 1452 of theTU 802 coefficients illustrated in FIG. 14A. Non-zero coefficients areindicated with a logical 1 and zero coefficients are indicated with alogical 0 (although a different mapping may be used).

Returning to the example of FIG. 14A, if the scan pattern is ahorizontal scan pattern 1302A, the last non-zero coefficient of the TU802 is coefficient 1404. This can also be seen in FIG. 14B, as thesignificance map element associated with coefficient 1404 is a logical 1and is the last element in the horizontal scan pattern 1202A.

Regardless of which scan technique or pattern is used, all quantizedcoefficients must be scanned, except those along the scan pattern thatare beyond the last coded coefficient of a TU 802. However, as HEVC iscurrently defined, it is possible that all of the coefficients of the TU802 will be scanned unless there is a reliable means for identifying andskipping the remainder of the coefficients after the last non-zerocoefficient in the scan pattern. Ideally, the means for identifying thelast non-zero coefficient should strike a balance between minimizingprocessing and the size of the resulting bitstream 314.

As described above with reference to the encoder 312 illustrated in FIG.3, HEVC implements context adaptive coding such as context adaptivebinary arithmetic coding (CABAC). CABAC is a form of entropy encodingthat encodes binary symbols using probability models. A non-binaryvalued symbol (such as a transform unit coefficient or motion vector) isbinarized or converted into a binary code prior to arithmetic coding.Stages are repeated for each bit (or “bin”) of the binarized symbol.

A context model is a probability model for one or more bins of thebinarized symbol. This model may be chosen from a plurality of availablemodels depending on the statistics of recently-coded data symbols. Thecontext model stores the probability of each bin being “1” or “0”. Anarithmetic coder then encodes each bin according to the selectedprobability model.

A context variable is a variable specified for the adaptive binaryarithmetic decoding process of a bin by an equation containing recentlydecoded bins. A cabac_init_flag specifies the method for determining theinitialization table used in the initialization process for contextvariables. The value of cabac_init_flag is in the range of 0 to 1,inclusive. When cabac_init_flag is not present, it is inferred to be 0.

The initialization of context variables is performed as follows. Foreach context variable, the two variables pStateIdx (probability stateindex) and valMPS (value of most probable symbol) are initialized. Froma table entry initValue, the two 4 bit variables (slopeIdx andintersecIdx) are derived according to the following pseudo-code process:

slopeIdx = initValue >> 4 intersecIdx = initValue & 15  Slope m andIntersec n are derived from the indices as follows: m = slopeIdx*5 − 45n = ( intersecIdx << 3 ) − 16

The last significant coefficient of a TU 802 in a scan pattern mayidentified by the variables last_significant_coeff_x andlast_significant_coeff_y.

FIG. 15A presents a table for determining the initialization variables mand n according to the last_significant_coeff_x andlast_significant_coeff_y variables. For 16×16 TU 802, the range of thevariables last_significant_coeff_x and last_significant_coeff_y is from0 to 15. The coefficients for the first half of the position range (0-7)are decoded using context based model as indicated. Six contexts areused for the luminescence (Luma) 16×16 TU. That is, ctxIdx 6-11 are forI-type intra coded pictures, ctxIdx 37-42 are for P-type inter codedpictures, and ctxIdx 68-73 are for B-type inter coded picture. FIG. 15Bpresents a table showing the ctxIdx offset for different CU 502 typed.

For 32×32 TU 802, the last_significant_coeff_x andlast_significant_coeff_y range from 0 to 31. The first half of theposition (range 0-15) are decoded using context based model in the tableshown in FIG. 15A. Seven contexts are used for a Luma 32×32 TU. That is,ctxIdx 12-18 are for I-type intra coded pictures, ctxIdx 43-49 are forP-type inter coded pictures, and ctxIdx 74-80 are for B-type inter-codedpictures. Further, the possible range of either thelast_significant_coeff_x or last_significant_coeff_y for rectangular TUs802 (32×8, 8×32, 16×4, and 4×16) is the value for the maximum of thelongest dimension of the TU 802. Hence, generally, the range of eitherlast_significant_coeff_x or last_significant_coeff_y could be from 0-31for any M×N TU where max(M,N) is larger than 16.

Scanning all of the TU 802 coefficients, even those with a value of zerocauses the encoder 312 to perform additional calculations and theencoder 312 to perform additional calculations and to send more bits inthe bitstream 314 than are necessary for the decoding process. Furtherexacerbating the problem, the probability of having a scan row, column,or diagonal with only zero coefficients may be higher with thenon-square TUs 802 introduced in HEVC.

Further, because of the large range (0-31) of last_significant_coeff_xand last_significant_coeff_y, the last coefficient position codingrequires a large number of bits to code two numbers from 0-31 and alarge number of contexts and bins to support multiple TU 802 sizes asshown in FIG. 15A.

Last Coefficient Coding and Decoding

One technique to reduce the size of the bitstream 314 and the binsnecessary for the coding of the quantized TU 802 coefficients is toencode a flag in the bitstream syntax that indicates when a particularportion of the TU 802 coefficients in a portion of the scan pattern hasat least one non-zero coefficient. Generally, this technique introducesa flag for a scan row, column or diagonal (for diagonal, maybe onlycertain size will have that flag), depending the scan pattern. If a scanrow, column, or diagonal has only zero coefficients, the flag is set tobe 0; otherwise, 1. This introduced flag uses fewer bits than directlyencoding the last significant coefficient of the transform unit, whilemaintaining adequate performance. To further reduce the overhead of flagbits, the number of set flags may be reduced by not setting any flagsfor remaining rows or columns in the scan pattern if they have all zerocoefficients. The flag bit overhead can be still further reduced becausethe last row or column may be inferred from the flag for the precedingrow or column. These features are described in more detail below.

FIG. 16A is diagram presenting an exemplary technique for performing alast coefficient coding technique that can be used in an HEVCencoding/decoding system 202. In block 1602, a determination is made asto whether all of the coefficients of a portion of the TU 802 disposedon a portion of a scan pattern in zero. Block 1604 sets a flagassociated with the portion of the TU 802 to a first value if at leastone of the coefficients of the portion of the TU 802 on the portion ofthe scan pattern is non-zero. Block 1606 sets a flag associated with theportion of the TU 802 to a second value if all of the coefficients ofthe portion of the TU 802 on the portion of the scan pattern are zero.Block 1608 determines if all of the scan portions have been considered.If not, processing passes back to block 1602 for consideration of thenext portion of the TU 802 coefficients on the next portion of the scanpattern. If all of the scan portions have been considered, processingfor the instant last coefficient processing for the TU underconsideration is completed, and processing for the next TU or other taskmay begin. Block 1610 is optional further embodiment that is furtherdiscussed below.

FIG. 17 is a diagram illustrating the operations of FIG. 16A. Forpurposes of simplicity, FIG. 17 discloses an embodiment using thehorizontal scan pattern 1302A illustrated in FIG. 13. In this scanpattern 1302A, a first portion of the TU 802 coefficients (e.g. those inthe first row) on the first portion 1702A of the scan pattern includesat least one non-zero coefficient, namely, the zeroth and fifthcoefficients of scan portion 1702A. Therefore, block 1604 sets firstflag 1706A associated with the first portion of TU 802 coefficients onthe first portion of the scan pattern 1702A to a logical 1.

Block 1608 determines that there are more scan portions remaining in thescan pattern (e.g. scan pattern portions 1702B-1702H), so processing isrouted back to block 1602, which again determines if there is at leastone non-zero coefficient the portion of the TU 802 transformcoefficients of the next scan pattern portion 1702B. In the illustratedexample, the second portion 1702 of the scan pattern 1302A also includesat least one non-zero coefficient (the second coefficient), hence, block1604 sets a second flag 1706B associated with the second portion 1702Bof the transform coefficients on a second portion of the scan pattern1302A.

Again, block 1608 determines that there are more scan portions1702C-1702H remaining in the scan pattern 1302A, so processing is routedto block 1602 and third portion 1702C of the coefficients of the TU 802are examined. In this case, all of the coefficients of the third portionof the TU on the third portion 1702C of the scan patter 1302A are zero,so block 1606 sets third flag 1706C to zero, indicating that thatportion of the scan pattern 1302A includes TU 802 coefficients that areall zero.

This process is repeated until all of the coefficients of the TU 802have been examined, and a flag has been set or not set to indicatewhether the associated scan portion 1702A-1702H (and portion of TU 802coefficients) have any non-zero values. In the illustrated embodiment,the since flag 1706A, 1706V, 1706D, and 1706F have a value of one, it isknown that at least one of these scan portion coefficients (in thiscase, rows) have a non-zero value. That the remaining flags have a valueof zero indicates that those scan portion coefficients all have a valueof zero.

Although the scan pattern 1302A for this example is a row-by-row canpattern for purposes of illustrative simplicity, the same technique canbe used for any scan pattern, including column-by-column scanning 1302C,either of the diagonal scan patterns 1302B and 1302D, wavefrontscanning, or even circular or non-continuous scan patterns. All that isrequired is that a flag or other artifice be set to a value indicatingthe zero/non-zero status of an associated portion of the TU 802coefficients on an associated portion of the scan pattern 1302.

The entropy coding of the TU 802 coefficients themselves may beperformed after each individual coefficient is examined to determinewhether it has a non-zero value, after the entire scan portion (e.g.row, column, or diagonal) has been examined to determine whether thereare any non-zero coefficients (for example, after block 1602 or block1606 of FIG. 16A) or after the entire TU 802 has been scanned todetermine whether there are any non zero coefficients (after theoperations shown in FIG. 16A have all been completed). By indicating thelast scan portion 1702F that includes a non-zero value, the foregoingparadigm allows the encoder and decoder to forego the decoding of TU 802coefficients that have zero values, thus saving bits in the bitstream314 and the associated processing required to decode the encoded TU 802coefficients. In one embodiment, the flags 1706 described above may beknown as last_TU_scan_portion flags, which are sent in the syntax andassociated with the TU 802.

Indicating the last non-zero coefficient in the scan pattern by a simplenumber or two numbers analogous to Cartesian coordinates (e.g.last_significant_coeff_x and last_significant_coeff_y) can also resultin bitstream 314 and processing savings. However, particularly with thelarger TUs made possible under HEVC, (which can be as large as 32×32), asignificant number of bits are wasted. The foregoing technique allowsthe processing of zero coefficients to be reduced, while substantiallyreducing the number of bits required in the bitstream 314 for lastcoefficient signaling.

While the foregoing reduces the number of zero value TU 802 coefficientsto be encoded, it still may result in some number of zero value TU 802coefficients to be encoded, for example, the last two coefficients inrow 1702F. This number may be further reduced by setting an additionalflag (for example, a last_scan_portion_position flag) associated onlywith the last scan portion that indicates that the particularcoefficient is the last coefficient 1704 in the last portion of the scanpattern to have a non-zero value.

Further, while the foregoing example used a flag as an indicator of theexistence of a non-zero coefficient in the scan portion, any analogousartifice can be used for such signaling. For example, an index or othervalue can be used.

The number of bits required to indicate that last scan portion can befurther reduced, as indicated by dashed block 1610 of FIG. 16A. Block1610 determines whether all of the coefficients of the remainingportions of the TU 802 are zero. If not (non-zero coefficients remain inthe remainder of TU 802 and scan portions), block 1612 allows processingto be routed to block 1604 to consider the next scan portion, as before.However, if all of the coefficients of the remaining portion of the TU802 are zero, coding of TU 802 under consideration is ceased.

Referring back to the example presented in FIG. 17, the processing ofscan portions 1702A-1702E proceed as before, and flags 1706A-1706E areset as indicated in the diagram. However, after flag 1706E is set,processing is routed to block 1608, which determines that there are moreportions of the scan pattern to be considered, and processing is therebyrouted back to block 1602. Block 1602 determines that there is at leastone non-zero coefficient in scan pattern portion 1702F (in fact, thereare two). Next, processing is passed to block 1610, which considerswhether the coefficients of all of the remaining portions of the TU 802(e.g. the coefficients in scan portions 1702G and 1702H are equal tozero. In the example shown in FIG. 17, all of the remaining portions ofthe TU 802 have zero coefficients, so block 1610 routes processing tolink A, which ends the last coefficient processing for the TU 802 underconsideration. The resulting set of last coefficient flags associatedwith the TU belong to a last coefficient flag set 1708 which nowexcludes flags associated with scan portions 1702G and 1702H. This savesthree bits of information otherwise used in the baseline embodiment.Note that this technique also saves an additional bit over simplyforegoing the addition of any trailing zero flags such as flags 1706Gand 1706H to the last coefficient flag set when the last two scanportions 1702G and 1702H have all zero coefficients. This is possiblebecause the information conveyed by the fact that flag 1706E is the lastflag in the set of last coefficient flags is sufficient to infer thatthe next row or portion of the scan is the last portion with a non-zeroTU coefficient.

FIG. 16B is a diagram illustrating the decoding of the encoded bitstream314, using the last coefficient coding. In block 1652, a flag 1706associated with all of the coefficients of a first portion of a TU 802disposed on a first portion of a scan pattern is read. If the flag 1706has the first value, the first portion of the TU 802 coefficientsdisposed in the first portion of the scan pattern are decoded, as shownin block 1654. If the flag 1706 has a second value, the coding of thefirst portion of TU 802 coefficients disposed on the first portion ofthe scan pattern 1702A is skipped, as shown in block 1656. Block 1662checks to determine if there are any more scan portions 1702 in the TU802. If there are additional scan portions 1702, processing is routedback to block 1652, whereas if there are no additional scan portions,decoding processing for the TU 802 under consideration is terminated.

For example, referring back to FIG. 17, flag 1706A indicates that thereis a non-zero coefficient in the first portion 1702A of the scanpattern. Accordingly, processing is routed to block 1654, and the firstportion of the TU coefficients (those in the scan row portion 1702 aredecoded. The process is completed for scan pattern portion 1702B becauseflag 1706 is also set to one, indicating that there is a non-zerocoefficient value in the TU 802 coefficients of the second scan patternportion 1702B. Flag 1706C indicates that all coefficients in scanpattern portion 1702C have the second value, and decoding of theassociated coefficients of the TU 802 in the scan portion 1702C isskipped. This process is repeated for the remaining flags 1706D-1706H.

Dashed blocks 1658 and 1662 refer decoding using the optional embodimentintroduced in block 1610 of FIG. 16A. Block 1658 determines whether theread flag is the last remaining flag of a flat set 1708. If either theread flag 1706 is not the last remaining flag of the flag set,processing is routed to block 1662 and is continues as previouslydescribed. But if the read flag is the last remaining flag of the flagset, the portion of the TU 802 coefficients associated with the portionof the scan pattern 1702 following the current scan portion 102associated with the remaining flag are decoded, and coding of theremaining coefficients of the TU 802 is skipped, as shown in block 1664.

For example, referring again to FIG. 17, the last remaining flag of theflag set 1708 is flag 1708E, which has a value of zero. Since this isthe last remaining flag of the flag set 1708, block 1664 decodes thecoefficients of the portion of the TU 802 following the portionassociated with the last remaining flag 1706E (which includes thecoefficients in scan pattern portion 1702F, and skips the decoding ofthe remaining coefficients in the TU 802 being decoded, including thecoefficients in scan portions 1702G and 1702H.

In the examples presented, the coding and decoding of the TU 802coefficients and the setting of the flags 1706 or 1708 was performedfollowing the scan pattern 1302A in the direction. As described in FIG.16B, the decoding of the coefficients can be performed in the samedirection, or may be coded in an opposite direction from the scanpattern.

For example, referring to FIG. 17, the decoding of the encoded TU 802coefficients can be performed by starting from scan portion 1702H, thelast scan portion. Flags 1706H and 1706G indicate that scan portions1702H and 1702G have all zero TU 802 coefficients, so the decoding ofthe coefficients in these scan pattern portions, if they are in thebitstream 314 at all, can be skipped. Flag 1706F indicates that scanpattern portion 1702F includes some non-zero coefficients, so thecoefficients of this portion of the TU 802 can be decoded. This processcan then continue for the remainder of the TU 802 coefficients. Or,logic can be inserted prior to block 1652 that determines the last flagassociated with the scan pattern that has the first value (e.g. 1706F),and simply begin decoding of the coefficients of the scan portionassociated with that flag (e.g. 1706F) and the remaining scan portionsof the TU 802 prior to that scan portion (e.g. 1702E-1702A).

Similarly, in the alternative embodiment shown in the dashed blocks ofFIGS. 16A and 16B, the last flag 1708E of the flag set 1708 indicatesthat the last scan pattern portion 1702 with a non-zero TU 802coefficient is the scan pattern portion that follows the scan patternportion associated with the last flag 1708, or scan pattern portion1702F. Decoding of the TU coefficients can therefore begin with the TU802 coefficients in scan pattern portion 1702F and end with scan patternportion 1702A.

The illustrations used to describe the foregoing used a row by row scanpattern such as is illustrated in 1302A of FIG. 13. It is noted,however, that the technique itself can be performed with a diagonal scanpattern or any scan pattern, so long as the appropriate flag isassociated with the portion of the scan pattern. It is also noted that arule can be imposed for diagonal scan patterns (or others in which thenumber of coefficients in the scan pattern vary from portion to portion)in which the flag indicating the zero/non-zero status of the in the scanportion is only used when the scan portion has a minimum number ofcoefficients.

Last Coefficient Group Position Encoding and Decoding

Another technique to reduce the size of the bitstream 314 and the binsnecessary for the coding of the quantized TU 802 coefficients is toencode a flag in the bitstream syntax that indicates when a particularportion or group of TU 802 coefficients in a portion of a scan patternhas at least one non-zero coefficient.

FIG. 18A is a diagram of 16×16 TU 802 having a plurality of coefficients1802, and FIG. 18B is a significance map generated from the TU 802illustrated in FIG. 18A. The significance map 1820 has a “1” in eachcoefficient position having a non-zero value.

The TU 802 shown in FIG. 18A may be scanned and encodedcoefficient-by-coefficient using a diagonal scan pattern 1824 analogousto the scan pattern of 1302A, as shown in FIG. 18C. Note that the lastnon-zero coefficient 1822 is located in the 12th position in the xdirection and the 10th position in the y direction. Since the remainingcoefficients are zero, they need not be entropy encoded, nor need theybe decoded by the decoder 258. By not encoding these coefficients, thebitstream 314 is correspondingly reduced and the processing required atboth the encoder 240 and decoder 258 can be reduced. To implement thisfunctionality, the last non-zero coefficient 1822 of the TU 802 in thescan pattern 1824 is indicated in the syntax associated with the TU 802in Cartesian coordinates using variables known aslast_significant_coeff_x and last_significant_coeff_y. These values arepassed in the bitstream 314 along to the decoder. In the example shownin FIG. 18C, last_significant_coeff_x=12 andlast_significant_coeff_y=10.

The scan pattern 1824 may also be performed in the opposite directionshown in FIG. 18C, but along the same path, beginning with the lastcoefficient of the scan path 1824. This permits the position of the lastsignificant coefficient 1822 to be determined more rapidly and byconsidering fewer coefficients.

The problem is that since the TU 802 can be as large as 32×32, thecoding of the last_significant_coeff_x and last_significant_coeff_yvariables requires a significant number of bits. To provide analogousfunctionality while requiring fewer bits, a two-level significance mapmay be adopted, which partitions TUs 802 larger than a specific size(for example, 16×16) into non-overlapping portions known as subblocks ofthe same size (e.g. 4×4).

FIG. 18D is a diagram illustrating a TU 802 that has been partitionedinto sixteen non-overlapping subblocks 1822A-1822P, each subblock 1822having the same 4×4 dimension and each including sixteen coefficients.The resulting partitioned TU 802 permits the definition of a multi-levelsignificance map coding for the TU 802 that includes a level zero (L0)significance map and level one (L1) significance map.

The L0 significance map indicates which coefficients within a TU 802 arenonzero coefficients. If a coefficient is a non-zero coefficient, asignificant_coeff_flag associated with the coefficient is set to a firstvalue, such as “1”. Otherwise, the significant_coeff_flag associatedwith that coefficient is set to a second value, such as “0”. FIG. 18B isa diagram showing an exemplary L0 significance map 1830 for the TU 802shown in FIG. 18A.

The L1 significance map indicates which 4×4 subblocks within a TU 802contain at least one non-zero coefficient. If a 4×4 subblock (forexample, subblock 1832A) has at least one nonzero coefficient, asignificant_coeffgroup_flag associated with that subblock is set to afirst value such as a “1”. Otherwise, (for example, subblock 1832P) thesignificant_coeffgroup_flag is set to a second value such as a “0”.

FIG. 18E is a diagram of an L1 significance map 1840 corresponding tothe partitioned TU 802 of FIG. 18D. Note that the dashed subblocks ofthe TU 802 illustrated in FIG. 18D (subblocks 1832D, 1832H, and1832M-1832P) each have all zero coefficients, and that this is reflectedby the significant_coeffgroup_flag associated with the related portionsof the L1 significance map 1840 having a value of “0”. Note also thatthe remaining subblocks have at least one non-zero coefficient and thussubblocks 1832A-1832C, 1832E-1832G, and 1832I-1832L each have anassociated significant_coeffgroup_flag with a value of “1”.

Also note that if the subblocks 1842A-1842P were scanned using thediagonal subblock scan pattern 1832 analogous to the scan pattern of1302A (e.g. diagonally and subblock-by-subblock) as shown in FIG. 18E,subblock 1832L is the last subblock in the subblock scan pattern of theTU 802 to have a non-zero coefficient therein. This subblock 1832L isknown as the last significant subblock 1832.

The position of the last significant subblock 1832L may be computed andcoded in the bitstream 314. In one embodiment, the last significantsubblock 1832L position is expressed in terms of the horizontal andvertical position from the upper leftmost subblock 1832A, usingvariables last_significant_subblock_coeff_x andlast_significant_subblock_coeff_y. In the exemplary L1 significance mapand scan pattern of FIG. 18E, (last_significant_subblock_coeff_x,last_significant_subblock_coeff_y) is (3,2). Using subblocks ofcoefficients rather than coefficients to define the position reduces therange of the stored variables from 0-3 to 0-15 for a 16×16 TU 802, andgenerating a L1 significance map only up to the last significantsubblock 1832L along the subblock scan pattern 1834 permits furthersavings.

The position of the last significant subblock 1832L can be expressed inother terms, for example, in the number of subblocks 1832 along thesubblock scan path 1834 from the first subblock in the scan pattern1832A.

FIG. 19 is a diagram presenting an exemplary technique for performing alast subblock coding technique that can be used in an HEVCencoding/decoding system 202. In block 1902, the last subblock 1832Lalong a subblock processing scan pattern 1834 that has at least onenon-zero TU coefficient (e.g. the last significant subblock) isdetermined. In block 1904, the position of the determined lastsignificant subblock 1832L is encoded. As described above, the positionof the last significant encoded subblock 1832L may be encoded aslast_significant_subblock_coeff_x and last_significant_subblock_coeff_yvalues, or by other syntax.

Turning next to block 1908, a first value is encoded for each subblockpreceding the last significant subblock in the subblock processingpattern that has at least one non-zero coefficient. For example, in FIG.18E, the last significant subblock 1832 is subblock 1832L, and the eachof the preceding subblocks in the scan pattern 1834 (e.g. subblocks1832A-1832O) are examined to determine which of those subblocks includea non-zero TU 802 coefficient. Subblocks having a non-zero TU 802coefficient (e.g. subblocks 1832A-1832C, 1832E-1832G, and 1832I-1832K)are encoded with a first value (in the illustrated example, “1”). Notethat the a value for the last significant subblock 1832L may also beencoded as “1” but need not be so, because that subblock, by definition,has a non-zero coefficient.

FIG. 18F is a diagram illustrating a modified L1 significance map (orsubblock significance map) generated using the foregoing paradigm. Notethat since subblock 1832P follows the last significant subblock 1832L inthe subblock scan path 1834, it is known that this subblock 1832P has nonon-zero coefficients, and there coding of a “1” or a “0” for this blockcan be skipped, thus saving the bits that may be used encode thisinformation. Also, note that a first value (“1”) is not coded for thelast significant subblock 1832L, as the fact that it includes non-zerocoefficients is already known. In alternate embodiments, a first valuemay be encoded for this subblock, if desired.

Block 1912 encodes the coefficients 1802 of the last significantsubblock 1832L and the coefficients of the subblocks 1832 preceding thelast significant subblock 1832L along the subblock scan pattern 1834having the first encoded value (“1”).

In the example shown in FIG. 18F, the coefficients of last significantsubblock 1832L are encoded and the coefficients of subblocks1832A-1832C, 1832E-1832G, and 1832I-1832K and 1832M-1832O along subblockscan path 1834 are also encoded. The encoding of subblocks 1832D, 1832H,1832M-1832O may be skipped (as shown in block 1914) as they are known toinclude no non-zero coefficients (since they are not associated with thefirst encoded value (“1”) but rather the second encoded value (“0”). Theencoding of block 1832P may also be skipped because it follows the lastsignificant subblock 1832L in the subblock scan pattern, and istherefore known not to have any non-zero coefficients.

The foregoing can be accomplished in order of the subblock scan pattern1834 (e.g. from the first subblock in the scan pattern 1832A to the lastsignificant subblock 1832L) or may be accomplished in reverse order(from the last significant subblock 1832L and proceeding to the firstsubblock 1832A in the subblock scan pattern 1834).

FIG. 20 presents a diagram illustrating the coefficients of a candidatesubblock having the first encoded value indicating that it includesnon-zero coefficients. One such subblock 1832 is subblock 1832A, whichis illustrated in the diagram. The coefficients 1802 of each subblockare scanned and encoded following a coefficient scan pattern 2002. Thecoefficient scan pattern 2002 illustrated in FIG. 20 follows ananalogous scan pattern to the subblock scan pattern 1834, but this notneed be the case, as the coefficient scan pattern 2002 may differ fromthe subblock scan pattern 1834. Further, although the coefficient scanpattern 2002 illustrated in FIG. 20 is indicated as being scanned fromthe top-leftmost coefficient 1802A to the lower rightmost coefficient1802P, this need not be the case, as the direction may be reversed.

The coding of the coefficients themselves may be performed as shown inFIG. 18G. In this embodiment, a significance map 1820 for thecoefficients of the subblock under consideration (in the illustratedcase, subblock 1832A) is generated, resulting in a corresponding portionof the significance map 1820 (illustrated as item 1820A) for the entireTU 820. The difference between the portion of the coefficients 1832A andthe significance map 1820A is obtained and used as the codedcoefficients 1850A. For example, the first row of the first subblock1832A has the values (5, 2, 0, 1). These corresponding significant map1820 (1, 1, 0, 1) values may be subtracted from the TU 820 values toproduce the values (4, 1, −, 0), and these resulting values may befurther coded. This allows the same information to be conveyed whileusing fewer bits. Note that the zero-valued coefficients of the subblock1832A are not encoded during this process, but the non-zero coefficientsare coded.

Note that the significance map 1820 for the TU 802 may be generated as apart of the encoding process. However, using the foregoing techniques,the significance map 1820 may be generated only for the last significantsubblock 1832L and the subblocks that precede it in the subblock scanpattern 1834.

The aforementioned technique can be applied only to TUs 802 exceeding aparticular size or dimension, for example, 16×16, 16×4, 4×16, 32×32,32×8 and 8×32. For a generalized M×N TU 802, coding the last significantsubblock position using the two-level significance map instead of thelast significant coefficient position:

-   -   1. Reduces the range of the last significant position: If a        16×16 TU 802 is divided into 4×4 subblocks, the range of the        last coded position is reduced from 0-15 for the last        significant coefficient values to 0-3 for 16×N or N×16 TU 802        for the last significant subblock values. Similarly, the range        of the last coded position is reduced from 0-31 to 0-7 for 32×N        or N×32 TUs 802, where N is equal to, or smaller less than, the        other dimension size.    -   2. Reduces the context table size for coding the syntax of the        last coded position: Instead of 6 luma contexts for 16×16 and 7        luma contexts for 32×32 of each type (I, P, B), only half size        of contexts, or even smaller than half size of contexts, are now        needed for coding of the last subblock position. Also, it is        possible that the contexts for 4×4 and 8×8 TU 802 may be used to        code the last coded subblock position of 16×16 and 32×32 TUs        802. In this case, the 16×16 and 32×32 contexts can be        completely removed. Finally, for an M×N TU 802 where M≠N and        max(M,N) is larger or equal to 16, the same benefits in context        reduction can also be possible.

Returning to FIG. 19, dashed blocks 1906, and 1910 indicate operationsthat may be optionally performed in addition to those already discussed.For example, in the foregoing embodiment, a first value “1” was encodedfor each subblock 1832 preceding the last significant subblock 1832L inthe scan pattern 1834 having a non-zero coefficient. Any subblockwithout an encoded first value is therefore known to not includecoefficients with non-zero values, and it is therefore unnecessary tocode information describing this fact. However, in one embodiment, asecond value (such as a “0”) can be encoded for each subblock 1832preceding the last significant subblock 1832L) having no non-zerocoefficients (or all zero coefficients). In this embodiment, thecoefficients of subblocks having the second value may be skipped.

FIG. 19 also refers to another optional embodiment in which the positionof the last non-zero coefficient 1822 of the last significant subblock1832L is also encoded, as shown in block 1906.

While coding according to the last significant subblock rather than thelast significant coefficient saves a number of bits, it also can resultin a number of bits trailing the last significant coefficient having azero value to be coded. For example, in FIG. 18E, the last significantsubblock is subblock 1832L. If all of the coefficients in this subblock1832L are coded (and subsequently decoded), a number of trailingcoefficients (e.g. the zero coefficients following the last non-zerocoefficient 1822) will be unnecessarily encoded and decoded.

FIG. 21 is a diagram of the last significant subblock 1832L of the TU802. Coefficient 1822 is the last significant coefficient on thecoefficient scan pattern 2002 of the last significant subblock 1832L,and hence, the coefficients that follow this coefficient 1822 in thecoefficient scan pattern are all zeros and need not be coded.Consequently, the encoding of these coefficients can be skipped to saveprocessing and bits. This can be implemented by determining the lastsignificant coefficient 1822 of the last significant subblock 1832L asthe last coefficient along a coefficient scan pattern 2002 within thelast subblock 1832L having a non-zero value, the encoding a position ofthe determined last significant coefficient of the last significantsubblock. In one embodiment, the last significant subblock coefficientvalue can be expressed in coefficient unit displacements away from theupper leftmost coefficient of the last significant subblock 1832L. Inthe example illustrated in FIG. 21, for example, the last significantsubblock coefficient position can be described aslast_significant_subblock_last_significant_coeff_x=0 andlast_significant_subblock_last_significant_coeff_y=1. Encoding of the TU802 coefficients of this last significant subblock 1832L can cease sothat the encoding of trailing zero coefficients are skipped.

Returning to FIG. 19 the process of determining the last subblock alonga subblock processing pattern that has at least one non-zero coefficientcan be accomplished in a number of ways. In one embodiment, this isaccomplished using the last_significant_coeff_x and last significantcoeff_y values available from previous operations in the encoder 312.These last_significant_coeff_x and last significant coeff_y values canbe used to calculate the last significant subblock 1832L position. Forexample, the last significant coefficient 1822 position of the TU 802can be associated with the last significant subblock 1832L in Cartesiancoordinates by dividing the coordinate of the last significantcoefficient 1822 by the associated subblock dimension for eachcoordinate direction and disregarding the remainder. In the exampleshown in FIG. 18D, for example, the last significant coefficient valuesare (12,10) and the subblock dimensions are 4×4. In this case, the lastsignificant subblock can be determined as (int[12/4], int[10/4])=(3,2),wherein the int[*] function returns the integer resulting from theoperation. If the last significant subblock is defined in other ways(e.g. the number of subblocks from the first subblock in the subblockscan pattern), the last subblock may still be mapped from the lastsignificant coefficient position, but more a more complex algorithm maybe required.

If the last_significant_coeff_x and last significant coeff_y values areunavailable to map into the position of the last significant subblockposition, they may be determined simply by scanning the coefficientsusing the array scan pattern 1824 or the same subblock scan pattern 1834and coefficient scan pattern 2002 that will be used in the decoder untilthe last non-zero coefficient is found. This may be advantageouslyperformed beginning at the last coefficient of the TU 802 in the scanpattern 1824 and scanning until a non-zero coefficient is found, or bybeginning at the first coefficient of the TU 802 and scanning the entireTU 802.

The technique described above can be practiced in other embodiments. Forexample, instead of determining the last significant subblock and onlycoding a first value for those subblocks preceding the last significantsubblock, the technique may be practiced by scanning the coefficients ofthe TU 802 and coding a first value for all subblocks having non-zerovalues (and optionally coding a second value for all subblocks havingonly zero values), and designating the last significant subblock as thelast subblock in the subblock scan pattern that has a coded first value.This technique is useful where a complete L1 level significance map isalready available, as the last significant subblock can be readilyascertained from a complete L1 significance map.

FIG. 22 is a diagram illustrating an exemplary technique for decoding TU802 coefficients that were encoded using the foregoing last coefficientgroup position technique described above. As described above, theencoding process creates a bitstream 314 that includes syntax for thelast significant subblock (e.g. last_significant_subblock_coeff_x andlast_significant_subblock_coeff_y). Block 2202 decodes a position of thelast significant subblock 1832L, which is the last subblock having atleast one non-zero coefficient along the subblock processing scanpattern 1834. Depending on how the last significant subblock is encoded,(e.g. as last_significant_subblock_coeff_x andlast_significant_subblock_coeff_y, the number of subblocks from thefirst subblock in the scan pattern, or using a row or column index asdescribed above) this may require computing the last significantsubblock position using the information passed in the syntax to accountfor different scan patterns.

Block 2206 decodes a first value (“1” for example) for each subblock1832 preceding the last subblock 1832L along the subblock processingscan pattern that has at least one non-zero coefficient. This can beaccomplished, for example, by looping through all of the subblocks 1832preceding the last subblock 1832L and decoding thesignificant_coeff_group flag associated each of the subblocks. Thisidentifies all subblocks 1832 preceding the last significant subblock1832L having non-zero coefficients. This can be performed along thesubblock scan pattern beginning at the last significant subblock 1832Land ending at the first subblock 1832A in the scan pattern, or from thefirst subblock 1832A in the scan pattern and ending at the lastsignificant subblock 1832L.

The coefficients of the last significant subblock 1832L and thesubblocks having the first encoded value (e.g. “1”) are then decoded, asshown in block 2210. Again, this can follow the subblock scan pattern1834 in either direction, and the scanning of the coefficients withineach subblock 1832 can be performed in the coefficient scan pattern2002, in either direction. As shown in block 2212, the decoding of thecoefficients within subblocks 1832 not having thesignificant_coeff_group flag with the first value may skipped, becauseit is known that the coefficients in these subblocks all have zerovalues. Further, the decoding of coefficients within subblocks thatfollow the last significant subblock 1832L in the subblock scan pattern1834 (e.g. subblock 1832P need not be performed, as it is known thatsuch subblocks do not include non-zero coefficients.

As described earlier, it can be inferred that any subblock 1832 nothaving an associated significant_coeff_group flag of the first value(“1”) may be assumed to have all zero coefficients. However in theembodiment described in FIG. 19, block 1910 encoded a second value foreach subblock preceding the last subblock in the subblock processingscan pattern that had all zero coefficients. Hence, the status of suchsubblocks 1832 is expressly described by the significant_coeff_groupflag having a second value (“0”). In this case, the decoding of thecoefficients of any subblock 1832 having an associatedsignificant_coeff_group flag having the second value is skipped, so thatonly those subblocks 1832 associated with the significant_coeff_groupflag having the first value will have their coefficients decoded. Thisis shown in block 2208 of FIG. 22.

Also as described in block 1906 of FIG. 19, the position of the lastsignificant coefficient within the last significant subblock 1832L maybe encoded using the last_significant_subblock_last_significant_coeff_xand last_significant_subblock_last_significant_coeff_y variables. Ifsuch information is encoded into the bitstream 314, block 2204 decodesthe information so that the position of the last significant coefficientof the last significant subblock is known. The decoding of thecoefficients of the last significant subblock 1832L can begin at thisposition (e.g. coefficient 1822) and proceed along the coefficient scanpath 2002 in a reverse direction to the first coefficient of the lastsignificant subblock 1832L or may begin at the first coefficient and endat the last significant coefficient 1822. Note that this is animprovement from coding and decoding the last significant coefficient ofthe entire TU 802 array, since the values describing last significantcoefficient within a 4×4 subblock will only need to code values between0-3 rather than 0-16 or 0-32.

While the foregoing has been discussed with respect to the setting offlags, indexes, or other designators to a particular value or state forpurposes of passing information or process signaling, the use of theterm flag and index is intended to broadly encompass the setting of anydesignator or other memory artifice for the purpose or reading thatdesignator or memory artifice for later use as described herein.

Hardware Environment

FIG. 23 illustrates an exemplary processing system 2300 that could beused to implement the embodiments of the invention. The computer 2302comprises a processor 2304 and a memory, such as random access memory(RAM) 2306. The computer 2302 is operatively coupled to a display 2322,which presents images such as windows to the user on a graphical userinterface 2318B. The computer 2302 may be coupled to other devices, suchas a keyboard 2314, a mouse device 2316, a printer, etc. Of course,those skilled in the art will recognize that any combination of theabove components, or any number of different components, peripherals,and other devices, may be used with the computer 2302.

Generally, the computer 2302 operates under control of an operatingsystem 2308 stored in the memory 2306, and interfaces with the user toaccept inputs and commands and to present results through a graphicaluser interface (GUI) module 2318A. Although the GUI module 2318A isdepicted as a separate module, the instructions performing the GUIfunctions can be resident or distributed in the operating system 2308,the computer program 2310, or implemented with special purpose memoryand processors. The computer 2302 also implements a compiler 2312 whichallows an application program 2310 written in a programming languagesuch as COBOL, C++, FORTRAN, or other language to be translated intoprocessor 2304 readable code. After completion, the application 2310accesses and manipulates data stored in the memory 2306 of the computer2302 using the relationships and logic that was generated using thecompiler 2312. The computer 2302 also optionally comprises an externalcommunication device such as a modem, satellite link, Ethernet card, orother device for communicating with other computers.

In one embodiment, instructions implementing the operating system 2308,the computer program 2310, and the compiler 2312 are tangibly embodiedin a computer-readable medium, e.g., data storage device 2320, whichcould include one or more fixed or removable data storage devices, suchas a zip drive, floppy disc drive 2324, hard drive, CD-ROM drive, tapedrive, etc. Further, the operating system 2308 and the computer program2310 are comprised of instructions which, when read and executed by thecomputer 2302, cause the computer 2302 to perform the steps necessary toimplement and/or use the invention. Computer program 2310 and/oroperating instructions may also be tangibly embodied in memory 2306and/or data communications devices 2330, thereby making a computerprogram product or article of manufacture. As such, the terms “articleof manufacture,” “program storage device” and “computer program product”as used herein are intended to encompass a computer program accessiblefrom any computer readable device or media.

The processing system 2300 may also be embodied in a desktop, laptop,tablet, notebook computer, personal data assistant (PDA), cellphone,smartphone, or any device with suitable processing and memorycapability. Further, the processing system 2300 may utilize specialpurpose hardware to perform some or all of the foregoing functionality.For example the encoding and decoding processes described above may beperformed by a special purpose processor and associated memory.

Those skilled in the art will recognize many modifications may be madeto this configuration without departing from the scope of the presentdisclosure. For example, those skilled in the art will recognize thatany combination of the above components, or any number of differentcomponents, peripherals, and other devices, may be used. For example,particular functions described herein can be performed by hardwaremodules, or a processor executing instructions stored in the form ofsoftware or firmware. Further, the functionality described herein can becombined in single modules or expanded to be performed in multiplemodules.

CONCLUSION

The foregoing description of the preferred embodiment has been presentedfor the purposes of illustration and description. It is not intended tobe exhaustive or to limit the disclosure to the precise form disclosed.Many modifications and variations are possible in light of the aboveteaching. It is intended that the scope of rights be limited not by thisdetailed description, but rather by the claims appended hereto.

What is claimed is:
 1. A method of processing a transform unit having anarray of coefficients, comprising: (a) determining if all of thecoefficients of a portion of the transform unit disposed on a portion ofa scan pattern are zero; (b) setting a flag associated with the portionof the transform unit to a first value if at least one of thecoefficients of the portion of the transform unit coefficients disposedon the portion of the scan pattern are non-zero by determining if all ofthe coefficients of the further portions of the transform unitcoefficients disposed on the scan pattern are zero, setting no furtherflags if all of the coefficients of the further portions of thetransform unit coefficients are zero, and setting the further flag onlyif at least one of the coefficients of the remaining portions of thetransform unit coefficients disposed on the remaining portion of thescan pattern is non-zero; (c) setting the flag associated with theportion of the transform unit coefficients to a second value if all ofthe coefficients of the portion of the transform unit coefficientsdisposed in the portion of the scan pattern are zero; (d) repeatingsteps (a)-(c) to set a further flag associated with each further portionof the transform unit coefficients disposed on each further portion ofthe scan pattern; and (e) encoding the flags and at least a portion ofthe transform unit coefficients into a coded bitstream.
 2. The method ofclaim 1, wherein encoding the flags the portion of the transform unitcoefficients into the coded bitstream includes encoding only thosecoefficients associated with a flag having the first value, and skippingencoding of those coefficients associated with a flag having the secondvalue.
 3. The method of claim 1, wherein the array is an N×M arraycomprising rows and columns and the scan pattern is a wavefront scanpattern that includes a plurality of diagonal scans in a diagonaldirection, and each portion of the scan pattern comprises all of thecoefficients in one diagonal of the plurality of diagonal scans of thewavefront scan pattern.
 4. A method of processing a transform unithaving an array of coefficients, comprising: (a) determining if all ofthe coefficients of a portion of the transform unit disposed on aportion of a scan pattern are zero, wherein the scan pattern is in a rowdirection, and each portion of the scan pattern comprises a row of aplurality of rows of coefficients; (b) setting a flag associated withthe portion of the transform unit to a first value if at least one ofthe coefficients of the portion of the transform unit coefficientsdisposed on the portion of the scan pattern are non-zero; (c) settingthe flag associated with the portion of the transform unit coefficientsto a second value if all of the coefficients of the portion of thetransform unit coefficients disposed in the portion of the scan patternare zero; and (d) encoding the flags and at least a portion of thetransform unit coefficients into a coded bitstream.
 5. A method ofprocessing a transform unit having an array of coefficients, comprising:(a) determining if all of the coefficients of a portion of the transformunit disposed on a portion of a scan pattern are zero, wherein the scanpattern is in a column direction, and each portion of the scan patterncomprises a column of a plurality of columns of coefficients; (b)setting a flag associated with the portion of the transform unit to afirst value if at least one of the coefficients of the portion of thetransform unit coefficients disposed on the portion of the scan patternare non-zero; and (c) setting the flag associated with the portion ofthe transform unit coefficients to a second value if all of thecoefficients of the portion of the transform unit coefficients disposedin the portion of the scan pattern are zero.
 6. A method of processing atransform unit having an array of coefficients, comprising: (a) readinga flag associated with all of the coefficients of a first portion of thetransform unit coefficients disposed on a first portion of a scanpattern, wherein each flag is a member of a last coefficient flag setassociated with the transform unit; (b) decoding the first portion ofthe transform unit coefficients disposed on the first portion of thescan pattern if the flag has a first value; and (c) skipping coding ofthe first portion of the transform unit coefficients disposed on thefirst portion of the scan pattern only if the flag has a second value,by determining if the flag is a last read flag of the last coefficientflag set and the last read flag has the second value, decoding theportion of the transform unit coefficients following the portion of thetransform unit coefficients associated with the last read flag andskipping decoding of the portions of further transform unitcoefficients; and (d) performing steps (a)-(c) for each remaining flagassociated with each remaining portion of the transform unitcoefficients of each remaining portion of the scan pattern.
 7. Themethod of claim 6, wherein the array is an n×m array comprising rows andcolumns and the scan pattern is a wavefront scan pattern in a diagonaldirection.
 8. The method of claim 6, wherein the array comprises aplurality of rows of coefficients, and the scan pattern is in a rowdirection.
 9. The method of claim 6, wherein the array comprises aplurality of columns of coefficients, and the scan pattern is in acolumn direction.
 10. A method of processing a transform unit having anarray of coefficients, comprising: setting a plurality of flags, eachflag associated with a portion of the transform unit coefficientsdisposed on a portion of a scan pattern, each flag set to a first valueif at least one of the coefficients in the associated portion of thetransform unit disposed on the associated portion of the scan pattern isnon-zero, and each flag set to a second value if all of the coefficientsin the associated portion of the transform unit disposed on theassociated portion of the scan pattern are zero, wherein the transformunit coefficients include a last non-zero coefficient disposed in a lastnon-zero coefficient portion of the scan pattern, the plurality of flagstogether comprise a flag set, and the flag set excludes flags associatedwith the portions of the scan pattern following the last non-zerocoefficient portion of the scan pattern; coding the flag set and theportions of the transform unit coefficients having an associated flaghaving the first value; and skipping coding of the portions of thetransform unit coefficients having an associated flag having the secondvalue.
 11. The method of claim 10, wherein: the flag set furtherexcludes the flag associated with the last non-zero coefficient portionof the scan pattern.
 12. A method of processing a transform unit havingan array of coefficients, comprising: reading a plurality of flags, eachflag associated with a portion of the transform unit coefficientsdisposed on a portion of a scan pattern, each flag set to a first valueif at least one of the coefficients in the associated portion of thetransform unit disposed on the associated portion of the scan pattern isnon-zero, and each flag set to a second value if all of the coefficientsin the associated portion of the transform unit disposed on theassociated portion of the scan pattern are zero, the transform unitcoefficients include a last non-zero coefficient disposed in a lastnon-zero coefficient portion of the scan pattern, the plurality of flagstogether comprise a flag set, and the flag set excludes flags associatedwith the portions of the scan pattern following the last non-zerocoefficient portion of the scan pattern; decoding the coefficients ofthe portions of the transform unit having an associated flag set to thefirst value; and skipping decoding the coefficients of the portions ofthe transform unit having an associated flag set to the second value.13. The method of claim 12, wherein: the flag set further excludes theflag associated with the last non-zero coefficient portion of the scanpattern.
 14. A system for processing a transform unit having an array ofcoefficients, comprising: a processor; and a memory, communicativelycoupled to the processor, the memory storing a plurality of instructionscomprising instructions for: (a) determining if all of the coefficientsof a portion of the transform unit disposed on a portion of a scanpattern are zero; (b) setting a flag from a last coefficient flag setassociated with the portion of the transform unit to a first value if atleast one of the coefficients of the portion of the transform unitcoefficients disposed on the portion of the scan pattern are non-zero bydetermining if all of the coefficients of the further portions of thetransform unit coefficients disposed on the scan pattern are zero,setting no further flags of the last coefficient flag set if all of thecoefficients of the further portions of the transform unit coefficientsare zero, and setting the further flag only if at least one of thecoefficients of the remaining portions of the transform unitcoefficients disposed on the remaining portion of the scan pattern isnon-zero; (c) setting the flag associated with the portion of thetransform unit coefficients to a second value if all of the coefficientsof the portion of the transform unit coefficients disposed in theportion of the scan pattern are zero; (d) repeating steps (a)-(c) to seta further flag associated with each remaining portion of the transformunit coefficients disposed on each remaining portion of the scanpattern; and (e) encoding the flags and at least a portion of thetransform unit coefficients into a coded bitstream.
 15. The system ofclaim 14 wherein encoding the flags and the portion of the transformunit coefficients into the coded bitstream includes encoding only thosecoefficients associated with a flag having the first value, and skippingencoding of those coefficients associated with a flag having the secondvalue.
 16. A system for processing a transform unit having an array ofcoefficients, comprising: a processor; and a memory, communicativelycoupled to the processor, the memory storing a plurality of instructionscomprising instructions for: (a) determining if all of the coefficientsof a portion of the transform unit disposed on a portion of a scanpattern are zero, wherein the scan pattern is in a row direction, andeach portion of the scan pattern comprises a row of a plurality of rowsof coefficients; (b) setting a flag associated with the portion of thetransform unit to a first value if at least one of the coefficients ofthe portion of the transform unit coefficients disposed on the portionof the scan pattern are non-zero; (c) setting the flag associated withthe portion of the transform unit coefficients to a second value if allof the coefficients of the portion of the transform unit coefficientsdisposed in the portion of the scan pattern are zero; and (d) encodingthe flags and at least a portion of the transform unit coefficients intoa coded bitstream.
 17. A system for processing a transform unit havingan array of coefficients, comprising: a processor; and a memory,communicatively coupled to the processor, the memory storing a pluralityof instructions comprising instructions for: (a) determining if all ofthe coefficients of a portion of the transform unit disposed on aportion of a scan pattern are zero, wherein the scan pattern is in acolumn direction, and each portion of the scan pattern comprises acolumn of a plurality of columns of coefficients; (b) setting a flagassociated with the portion of the transform unit to a first value if atleast one of the coefficients of the portion of the transform unitcoefficients disposed on the portion of the scan pattern are non-zero;(c) setting the flag associated with the portion of the transform unitcoefficients to a second value if all of the coefficients of the portionof the transform unit coefficients disposed in the portion of the scanpattern are zero; and (d) encoding the flags and at least a portion ofthe transform unit coefficients into a coded bitstream.
 18. A system forprocessing a transform unit having an array of coefficients, comprising:a processor; and a memory, communicatively coupled to the processor, thememory storing a plurality of instructions comprising instructions for:(a) determining if all of the coefficients of a portion of the transformunit disposed on a portion of a scan pattern are zero, wherein the arrayis an N×M array comprising rows and columns and the scan pattern is awavefront scan pattern that includes a plurality of diagonal scans in adiagonal direction, and each portion of the scan pattern comprises allof the coefficients in one diagonal of the plurality of diagonal scansof the wavefront scan pattern; (b) setting a flag associated with theportion of the transform unit to a first value if at least one of thecoefficients of the portion of the transform unit coefficients disposedon the portion of the scan pattern are non-zero; (c) setting the flagassociated with the portion of the transform unit coefficients to asecond value if all of the coefficients of the portion of the transformunit coefficients disposed in the portion of the scan pattern are zero;and (d) encoding the flags and at least a portion of the transform unitcoefficients into a coded bitstream.
 19. A system for processing atransform unit having an array of coefficients, comprising: a processor;and a memory, communicatively coupled to the processor, the memorystoring a plurality of instructions comprising instructions for: (a)reading a flag associated with all of the coefficients of a firstportion of the transform unit coefficients disposed on a first portionof a scan pattern, wherein each flag is a member of a last coefficientflag set associated with the transform unit; (b) decoding the firstportion of the transform unit coefficients disposed on the first portionof the scan pattern only if the flag has a first value; and (c) skippingcoding of the first portion of the transform unit coefficients disposedon the first portion of the scan pattern if the flag has a second valueby determining if the read remaining flag is a last read flag of thelast coefficient flag set and the last read flag has the second value,decoding the portion of the transform unit coefficients following theportion of the transform unit coefficients associated with the last readflag, and skipping decoding of the portions of further transform unitcoefficients; and (d) performing steps (a)-(c) for each remaining flagassociated with each remaining portion of the transform unitcoefficients of each remaining portion of the scan pattern.
 20. Thesystem of claim 19, wherein the array is an n×m array comprising rowsand columns and the scan pattern is a wavefront scan pattern in adiagonal direction.
 21. The system of claim 19, wherein the arraycomprises a plurality of rows of coefficients, and the scan pattern isin a row direction.
 22. The system of claim 19, wherein the arraycomprises a plurality of columns of coefficients, and the scan patternis in a column direction.
 23. A system of processing a transform unithaving an array of coefficients, comprising: a processor; and a memory,communicatively coupled to the processor, the memory storing a pluralityof instructions comprising instructions for: setting a plurality offlags, each flag associated with a portion of the transform unitcoefficients disposed on a portion of a scan pattern, each flag set to afirst value if at least one of the coefficients in the associatedportion of the transform unit disposed on the associated portion of thescan pattern is non-zero, and each flag set to a second value if all ofthe coefficients in the associated portion of the transform unitdisposed on the associated portion of the scan pattern are zero, whereinthe transform unit coefficients include a last non-zero coefficientdisposed in a last non-zero coefficient portion of the scan pattern, theplurality of flags together comprise a flag set, and the flag setexcludes flags associated with the portions of the scan patternfollowing the last non-zero coefficient portion of the scan pattern;coding the portions of the transform unit coefficients having anassociated flag having a the first value; skipping coding of theportions of the transform unit coefficients having an associated flaghaving the second value; and encoding the flag set and at least aportion of the transform unit coefficients into a coded bitstream. 24.The system of claim 23, wherein: the flag set further excludes the flagassociated with the last non-zero coefficient portion of the scanpattern.
 25. A system for processing a transform unit having an array ofcoefficients, comprising: a processor; and a memory, communicativelycoupled to the processor, the memory storing a plurality of instructionscomprising instructions for: reading a plurality of flags, each flagassociated with a portion of the transform unit coefficients disposed ona portion of a scan pattern, each flag set to a first value if at leastone of the coefficients in the associated portion of the transform unitdisposed on the associated portion of the scan pattern is non-zero, andeach flag set to a second value if all of the coefficients in theassociated portion of the transform unit disposed on the associatedportion of the scan pattern are zero, wherein the transform unitcoefficients include a last non-zero coefficient disposed in a lastnon-zero coefficient portion of the scan pattern, the plurality of flagstogether comprise a flag set, and the flag set excludes flags associatedwith the portions of the scan pattern following the last non-zerocoefficient portion of the scan pattern; decoding the coefficients ofthe portions of the transform unit having an associated flag set to thefirst value; and skipping decoding the coefficients of the portions ofthe transform unit having an associated flag set to the second value.26. The system of claim 25, wherein: the flag set further excludes theflag associated with the last non-zero coefficient portion of the scanpattern.